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ABSTRACT 


The  objective  of  the  effort  was  to  build  a  parallel  sparse  matrix  library  that  is  needed  by 
applications  migrating  to  parallel  computers.  Research  on  robust  algorithms  for  industrial 
problems  and  software  development  for  a  robust  implementation  for  industrial  users  are  both 
needed  for  a  sparse  matrix  library  in  order  for  parallel  computers  to  realize  their  potential  to 
out-perform  previous  single-threaded  hardware  and  software  platforms.  An  additional 
challenge  faced  during  the  effort  was  to  develop  software  that  would  run  efficiently  on  a 
variety  of  parallel  computers,  including  both  shared  memory  and  distributed  memory 
computers.  The  effort  included  work  on  the  solution  of  sparse  systems  of  linear  equations  in 
parallel  and  work  on  computing  some  eigenvalues  for  sparse  real  symmetric  generalized 
eigenvalue  problems.  These  technology  areas  are  key  for  the  migration  of  industrial 
applications  such  as  Finite  Element  Analysis  to  parallel  computers. 

The  effort  included  new  and  innovative  research  on  parallel  methods  for  solution  of  sparse 
systems  of  linear  equations.  A  software  package,  Sparse  Object  Oriented  Linear  Equation 
Solver  (SPOOLES),  containing  all  of  the  developed  software  for  solving  real  and  complex 
system  using  both  direct  and  iterative  methods  for  serial,  shared  memory,  and  distributed 
memory  computers  is  now  available  in  the  public  domain  via 
http://www.netlib.org/linalg/spooles/spooles.html. 

A  parallel  eigensolver  for  parallel  computers  was  also  developed  and  successfully 
demonstrated  on  several  test  problems  across  a  range  of  computers  (Sun,  SGI,  and  HP). 
Although  the  amount  of  code  that  was  actually  parallelized  is  substantial,  it  is  not  sufficient  to 
replace  the  serial  implementation  in  Fortran.  In  addition,  the  parallel  eigensolver  developed 
during  the  effort  requires  that  much  of  the  data  be  replicated  on  each  processor  instead  of 
being  fully  distributed.  This  leads  to  significant  limitations  on  problem  size  capacity,  which  in 
turn  limits  the  applicability  of  the  software.  Further  code  development  is  therefore  required 
for  a  sparse  real  symmetric  generalized  parallel  eigensolver  that  can  be  used  by  application 
programs  such  as  NASTRAN  and  ANSYS. 
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ACRONYMS,  ABBREVIATIONS  AND  SYMBOLS 


DARPA 

SPOOLES 


Defense  Advanced  Research  Programs  Agency 
Sparse  Object  Oriented  Linear  Equation  Solver 


1.0  SUMMARY 


The  Algorithmic  Research  and  Software  Development  for  an  Industrial  Strength  Sparse 
Matrix  Library  for  Parallel  Computers  performed  algorithmic  research  on  issues  associated 
with  the  parallel  solution  of  sparse  systems  of  linear  equations.  We  developed  a  new  set  of 
sparse  matrix  ordering  algorithms  to  both  preserve  sparsity  for  the  numerical  factorization, 
preserve  locality  for  efficient  local  computation  and  to  reduce  communication  and  enhance 
opportunities  for  parallel  computations.  We  implemented  a  suite  of  tools  for  the  direct 
solution  of  sparse  systems  of  linear  equations  supporting  both  real  and  complex  matrices, 
both  symmetric  and  unsymmetric  systems,  and  providing  pivoting  for  numerical  stability. 
Based  on  the  direct  solution  capabilities,  we  built  a  drop  tolerance  incomplete  factorization 
and  a  suite  of  iterative  methods  to  complement  the  direct  solution  capabilities.  We  also 
developed  a  capability  for  the  parallel  solution  of  large  least  squares  problems. 

We  also  performed  research  work  and  preliminary  software  development  for  a  parallel  real 
sparse  generalized  eigensolver.  While  leveraging  off  our  existing  serial  eigensolver  was  a 
viable  approach,  converting  the  serial  eigensolver  to  the  parallel  computing  environment 
was  significantly  more  difficult  than  originally  contemplated,  and  the  developed  software 
requires  further  research  and  development  to  improve  both  its  reliability  and  parallel 
efficiency. 

2.0  INTRODUCTION 

The  Computational  Mathematics  staff  of  the  Applied  Research  and  Technology  Division  of 
Boeing  Shared  Services  Group  is  responsible  for  mathematical  software  libraries  used 
throughout  The  Boeing  Company.  This  staff  is  also  responsible  for  the  commercially 
available  software,  BCSLIB-EXT,  which  is  used  extensively  inside  and  outside  of  Boeing. 
In  particular,  it  has  become  the  software  of  choice  for  solving  large  sparse  linear  algebra 
equations  arising  in  application  areas  such  as  finite  element  analysis.  The  Algorithmic 
Research  and  Software  Development  for  an  Industrial  Strength  Sparse  Matrix  Library  for 
Parallel  Computers  contract  sought  to  develop  the  basic  technology  for  such  a  library  for 
parallel  computers  and  to  make  that  technology  available  to  those  applications  attempting 
to  migrate  to  parallel  computers.  The  lack  of  this  enabling  software  technology  has  been  a 
large  stumbling  block  for  the  wide  spread  use  of  parallel  computers  for  large-scale 
applications. 

The  solution  of  sparse  linear  algebra  problems  in  parallel  has  been  an  active  area  of 
research  since  the  advent  of  parallel  computers.  However  none  of  this  research  has 
produced  a  viable  suite  of  software  for  the  parallel  solution  of  linear  equations  that  meets 
the  needs  of  industrial  applications. 

This  suite  of  software  is  required  to  extend  the  applicability  of  parallel  computers  to  general 
use  for  application  areas  that  have  to  solve  sparse  linear  algebra  problems  such  as  finite 
element  analysis.  This  contract  was  an  attempt  to  meet  this  need. 
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3.0  METHODS,  ASSUMPTIONS,  AND  PROCEDURES 


The  research  in  this  contract  drew  upon  Boeing's  experience  supporting  a  commercial 
software  library  in  this  application  area  and  knowledge  of  parallel  sparse  linear  algebra. 

We  identified  the  key  research  areas  that  required  breakthroughs  before  a  reliable  industrial 
strength  sparse  matrix  package  could  be  developed.  These  areas  were  ordering  methods, 
pivoting  for  numerical  stability,  and  a  robust  and  reliable  direct  factorization  approach  that 
would  work  well  in  serial,  shared  memory,  and  distributed  memory  parallel  computing 
environments. 

We  selected  a  development  environment  using  the  C  language,  with  POSIX  threads  for  the 
shared  memory  computing  paradigm  and  MPI  for  the  distributed  memory  computing 
paradigm.  We  discussed  interface  issues  with  technical  representatives  from  computing 
hardware  vendors  and  leading  providers  of  engineering  analysis  application  packages  that 
were  target  end  users  of  the  results  of  the  contract. 

We  enlisted  the  services  of  Dr.  Joseph  W.  H.  Liu,  York  University,  to  assist  in  the  first  task, 
which  was  the  development  of  a  new  ordering  methodology  for  the  direct  solution  of  sparse 
systems  of  linear  equations. 

Our  starting  point  for  the  parallel  eigensolver  was  the  block  shift  and  invert  Lanczos 
software  found  in  BCSLIB-EXT. 

4.0  RESULTS  AND  DISCUSSION 

During  the  course  of  this  contract  we  have  developed  a  robust  and  reliable  software 
package,  SPOOLES,  for  the  solution  of  sparse  systems  of  linear  equations  which  include 
the  following  features: 

•  Supports  both  Real  and  complex  sparse  matrices 

•  Solves  systems  that  are  Symmetric,  unsymmetric,  Hermitian,  or  overdetermined 

•  Provides  pivoting  for  numerical  stability 

•  Direct  and  iterative  solution  methods 

•  Available  on  serial,  shared  memory  parallel,  and  distributed  memory  parallel  computers 

SPOOLES  is  available  in  the  public  domain  via  NETLIB,  a  numerical  computing  software 
repository  at  Oak  Ridge  National  Laboratory.  Both  source  code  and  documentation  for 
SPOOLES  is  available  on  the  World  Wide  Web  via 
www.netlib.org/linal  g/spooles/spooles.2.2.html. 

The  sparse  eigensolver  developed  during  this  research  program  requires  further  research 
and  software  development  before  it  can  be  used  on  a  widespread  basis  as  a  sparse  real 
symmetric  generalized  parallel  eigensolver.  Boeing  is  pursuing  additional  funding  from 
alternate  sources  to  complete  this  work. 
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5.0  RECOMMENDATIONS 


We  recommend  that  any  application  developer  interested  in  solving  sparse  systems  of  linear 
equations  examine  the  SPOOLES  package  for  their  use. 
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